19 research outputs found
Scalable Inference of Symbolic Adversarial Examples
We present a novel method for generating symbolic adversarial examples: input
regions guaranteed to only contain adversarial examples for the given neural
network. These regions can generate real-world adversarial examples as they
summarize trillions of adversarial examples.
We theoretically show that computing optimal symbolic adversarial examples is
computationally expensive. We present a method for approximating optimal
examples in a scalable manner. Our method first selectively uses adversarial
attacks to generate a candidate region and then prunes this region with
hyperplanes that fit points obtained via specialized sampling. It iterates
until arriving at a symbolic adversarial example for which it can prove, via
state-of-the-art convex relaxation techniques, that the region only contains
adversarial examples. Our experimental results demonstrate that our method is
practically effective: it only needs a few thousand attacks to infer symbolic
summaries guaranteed to contain adversarial examples
Optimization and Abstraction: A Synergistic Approach for Analyzing Neural Network Robustness
In recent years, the notion of local robustness (or robustness for short) has
emerged as a desirable property of deep neural networks. Intuitively,
robustness means that small perturbations to an input do not cause the network
to perform misclassifications. In this paper, we present a novel algorithm for
verifying robustness properties of neural networks. Our method synergistically
combines gradient-based optimization methods for counterexample search with
abstraction-based proof search to obtain a sound and ({\delta}-)complete
decision procedure. Our method also employs a data-driven approach to learn a
verification policy that guides abstract interpretation during proof search. We
have implemented the proposed approach in a tool called Charon and
experimentally evaluated it on hundreds of benchmarks. Our experiments show
that the proposed approach significantly outperforms three state-of-the-art
tools, namely AI^2 , Reluplex, and Reluval
Bounded Expectations: Resource Analysis for Probabilistic Programs
This paper presents a new static analysis for deriving upper bounds on the
expected resource consumption of probabilistic programs. The analysis is fully
automatic and derives symbolic bounds that are multivariate polynomials of the
inputs. The new technique combines manual state-of-the-art reasoning techniques
for probabilistic programs with an effective method for automatic
resource-bound analysis of deterministic programs. It can be seen as both, an
extension of automatic amortized resource analysis (AARA) to probabilistic
programs and an automation of manual reasoning for probabilistic programs that
is based on weakest preconditions. As a result, bound inference can be reduced
to off-the-shelf LP solving in many cases and automatically-derived bounds can
be interactively extended with standard program logics if the automation fails.
Building on existing work, the soundness of the analysis is proved with respect
to an operational semantics that is based on Markov decision processes. The
effectiveness of the technique is demonstrated with a prototype implementation
that is used to automatically analyze 39 challenging probabilistic programs and
randomized algorithms. Experimental results indicate that the derived constant
factors in the bounds are very precise and even optimal for many programs
Privacy Risks of Securing Machine Learning Models against Adversarial Examples
The arms race between attacks and defenses for machine learning models has
come to a forefront in recent years, in both the security community and the
privacy community. However, one big limitation of previous research is that the
security domain and the privacy domain have typically been considered
separately. It is thus unclear whether the defense methods in one domain will
have any unexpected impact on the other domain.
In this paper, we take a step towards resolving this limitation by combining
the two domains. In particular, we measure the success of membership inference
attacks against six state-of-the-art defense methods that mitigate the risk of
adversarial examples (i.e., evasion attacks). Membership inference attacks
determine whether or not an individual data record has been part of a model's
training set. The accuracy of such attacks reflects the information leakage of
training algorithms about individual members of the training set. Adversarial
defense methods against adversarial examples influence the model's decision
boundaries such that model predictions remain unchanged for a small area around
each input. However, this objective is optimized on training data. Thus,
individual data records in the training set have a significant influence on
robust models. This makes the models more vulnerable to inference attacks.
To perform the membership inference attacks, we leverage the existing
inference methods that exploit model predictions. We also propose two new
inference methods that exploit structural properties of robust models on
adversarially perturbed data. Our experimental evaluation demonstrates that
compared with the natural training (undefended) approach, adversarial defense
methods can indeed increase the target model's risk against membership
inference attacks.Comment: ACM CCS 2019, code is available at
https://github.com/inspire-group/privacy-vs-robustnes
ReluDiff: Differential Verification of Deep Neural Networks
As deep neural networks are increasingly being deployed in practice, their
efficiency has become an important issue. While there are compression
techniques for reducing the network's size, energy consumption and
computational requirement, they only demonstrate empirically that there is no
loss of accuracy, but lack formal guarantees of the compressed network, e.g.,
in the presence of adversarial examples. Existing verification techniques such
as Reluplex, ReluVal, and DeepPoly provide formal guarantees, but they are
designed for analyzing a single network instead of the relationship between two
networks. To fill the gap, we develop a new method for differential
verification of two closely related networks. Our method consists of a fast but
approximate forward interval analysis pass followed by a backward pass that
iteratively refines the approximation until the desired property is verified.
We have two main innovations. During the forward pass, we exploit structural
and behavioral similarities of the two networks to more accurately bound the
difference between the output neurons of the two networks. Then in the backward
pass, we leverage the gradient differences to more accurately compute the most
beneficial refinement. Our experiments show that, compared to state-of-the-art
verification tools, our method can achieve orders-of-magnitude speedup and
prove many more properties than existing tools.Comment: Extended version of ICSE 2020 paper. This version includes an
appendix with proofs for some of the content in section 4.
DeepSearch: A Simple and Effective Blackbox Attack for Deep Neural Networks
Although deep neural networks have been very successful in
image-classification tasks, they are prone to adversarial attacks. To generate
adversarial inputs, there has emerged a wide variety of techniques, such as
black- and whitebox attacks for neural networks. In this paper, we present
DeepSearch, a novel fuzzing-based, query-efficient, blackbox attack for image
classifiers. Despite its simplicity, DeepSearch is shown to be more effective
in finding adversarial inputs than state-of-the-art blackbox approaches.
DeepSearch is additionally able to generate the most subtle adversarial inputs
in comparison to these approaches
Towards interpreting recurrent neural networks through probabilistic abstraction
National Research Foundation (NRF) Singapore under its AI Singapore Programm
Reactive probabilistic programming
International audienceSynchronous modeling is at the heart of programming languages like Lustre, Esterel, or SCADE used routinely for implementing safety critical control software, e.g., fly-bywire and engine control in planes. However, to date these languages have had limited modern support for modeling uncertainty-probabilistic aspects of the software's environment or behavior-even though modeling uncertainty is a primary activity when designing a control system. In this paper we present ProbZelus the first synchronous probabilistic programming language. ProbZelus conservatively provides the facilities of a synchronous language to write control software, with probabilistic constructs to model uncertainties and perform inference-in-the-loop. We present the design and implementation of the language. We propose a measure-theoretic semantics of probabilistic stream functions and a simple type discipline to separate deterministic and probabilistic expressions. We demonstrate a semantics-preserving compilation into a first-order functional language that lends itself to a simple presentation of inference algorithms for streaming models. We also redesign the delayed sampling inference algorithm to provide efficient streaming inference. Together with an evaluation on several reactive applications, our results demonstrate that ProbZelus enables the design of reactive probabilistic applications and efficient, bounded memory inference